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In open online communities such as e-commerce, participants need to rely on services provided by others in 
order to thrive. Accurately estimating the trustworthiness of a potential interaction partner is vital to a 
participant's well-being. It is generally recognized in the research community that third-party testimony sharing 
is an effective way for participants to gain knowledge about the trustworthiness of potential interaction partners 
without having to incur the risk of actually interacting with them. However, the presence of biased testimonies 
Keywords: adversely affects a participant's long term well-being. Existing trust computational models often require compli- 
Trust cated manual tuning of key parameters to combat biased testimonies. Such an approach heavily involves subjec- 


Reputation tive judgments and adapts poorly to changes in an environment. In this study, we propose the Actor-Critic Trust 
Credibility (ACT) model, which is an adaptive trust evidence aggregation model based on the principles of reinforcement 
Collusion learning. The proposed method dynamically adjusts the selection of credible witnesses as well as the key param- 


eters associated with the direct and indirect trust evidence sources based on the observed benefits received by 
the trusting entity. Extensive simulations have shown that the ACT approach significantly outperforms existing 
approaches in terms of mitigating the adverse effect of biased testimonies. Such a performance is due to the 
proposed accountability mechanism that enables ACT to attribute the outcome of an interaction to individual wit- 
nesses and sources of trust evidence, and adjust future evidence aggregation decisions without the need for human 
intervention. The advantage of the proposed model is particularly significant when service providers and wit- 
nesses strategically collude to improve their chances of being selected for interaction by service consumers. 

© 2014 Elsevier B.V. All rights reserved. 


1. Introduction * service providers (SPs), who provide services, goods or information re- 
quested by others and do not need to rely on others to perform these 


In open online communities where users are from diverse back- services; and 


grounds and may have conflicting interest, trust-based interaction deci- 
sion support is needed to sustain long term interactions among them. 
Nowadays, such systems are quite common (e.g., service oriented com- 
puting systems [1], e-commerce systems [2], wireless communication 
networks [3], and multi-agent systems [4] etc.). In such environments 
in which services and devices usually have limited capabilities, users 
often have to interact with each other in order to complete complex 
tasks. These interactions usually involve an exchange of services, infor- 
mation, or goods with value. Selfish users may renege on their commit- 
ments, thereby breaching the trust placed in them by others. Therefore, 
trust and reputation management mechanisms are often used to mini- 
mize the negative impact of selfish users. 


1.1. Background 


Generally, users in an open online community that can be modeled 
as multi-agent systems (MASs) may play two types of roles [1]: 
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* service consumers (SCs), who need to rely on service providers to 
accomplish certain tasks. 


The main objective of evidence-based trust models is to estimate the 
trustworthiness of a potential interaction partner which represents its 
true behavior pattern. Evidences about a service provider from the per- 
spective of a service consumer are usually from two sources: 


* direct trust evidence: which consists of a service consumer's direct in- 
teraction experience with the service provider; and 

* indirect trust evidence: which consists of third-party testimonies about 
the service provider from other service providers in the system. 


In practical systems, it is not possible to definitively know the trust- 
worthiness of a service provider. Therefore, it is often estimated using 
trust evidences. The estimation of a service provider's trustworthiness 
derived from the direct trust evidence of a service consumer alone is 
called direct trust, while that derived from the indirect trust evidence 
is called indirect trust. According to [4], an estimation derived from 
both sources of trust evidence is commonly known as the reputation 
of a service provider. In the eyes of a service consumer, other service 
consumers who provide it with indirect trust evidence (i.e. testimonies) 
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about a service provider are regarded as witnesses. A witness's reliability 
in terms of providing useful testimonies is referred to as its credibility. 

Since such systems tend to be very large in practice, service con- 
sumers often have to interact with service providers with whom they 
may not be very familiar (i.e. have little or no prior interaction experi- 
ence with) [5]. Thus, it is both necessary and advantageous to allow ser- 
vice consumers to act as witnesses to provide their own first-hand 
interaction experience as testimonies to other service consumers who 
lack such information. However, such an approach is not without its 
perils. 

Third-party testimonies may be biased and, thus, degrade the 
accuracy of trust decisions [1]. Therefore, testimonies from witnesses 
need to be filtered before being used to evaluate a service consumer's 
reputation. 

To this end, a number of evidence-based trust and reputation man- 
agement (TRM) models have been proposed over the years. The general 
flow for a service consumer to decide which service provider to select 
for interaction is illustrated in Fig. 1. Each service consumer continuous- 
ly records its direct interaction experience with service providers over 
time. When a service provider's trustworthiness needs to be evaluated, 
the service consumer may request third-party testimonies from wit- 
nesses, depending on the service consumer's confidence on its own 
direct trust evidence. These testimonies are preprocessed in an attempt 
to filter out unfair ratings. The resulting direct and indirect trust evi- 
dences are then aggregated to form a trustworthiness evaluation for 
that particular service provider. At the end of this process, the service 
consumer decides which service provider to interact with based on 
their trustworthiness evaluations. 


1.2. Research objectives 


Existing approaches for third-party testimony filtering and aggrega- 
tion commonly involve a crucial step in which the weight assigned to 
each third-party testimony and the weight assigned to the direct and 
the indirect sources of trust evidence need to be determined [6-9]. 

However, existing approaches often require manual tuning of key 
parameters in their models which heavily involves subjective judg- 
ments and adapts poorly to changes in the environment. 

In this paper, we address this limitation by proposing the Actor-Critic 
Trust (ACT) model based on the principles of the Actor-Critic Learning 
Method [10]. The ACT approach automates the adjustment of key thresh- 
old based parameters to eliminate human subjectivity and enhance the 
effectiveness of existing reputation evaluation models. Specifically, it 
enables existing evidence-based trust models to dynamically make two 
important decisions when presented with third-party testimonies for 
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a service provider: 1) how much weight to assign to its own personal 
direct trust evidence and the collective opinions from witnesses, and 
2) how much weight to assign to the testimonies from each witness. 
Experimental results, presented in Section 4, show that the ACT approach 
outperforms state-of-the-art approaches by around 20% in terms of 
improving the accuracy of finding trustworthy service providers in the 
presence of biased testimonies, especially when witnesses collude with 
malicious service providers. 

The rest of the paper is organized as follows. Section 2 reviews related 
work. Section 3 presents the basic notations used in this paper and the 
details of the proposed ACT approach. Section 4 describes the simulation 
test-bed and analyzes the results. The implications of the proposed ap- 
proach for practical decision support in online product review systems 
are discussed in Section 5. Finally, Section 6 presents a summary of our 
contributions and possible future work. 


2. Related work 


It is widely recognized within the research community that the im- 
portance of incorporating mechanisms to mitigate the adverse effects of 
biased testimonies. In this section, we discuss some recent research 
work on aggregating trust evidence from different sources and filtering 
out biased testimonies. For a more comprehensive review of this field, 
readers may refer to [1-3]. 


2.1. Trust evidence aggregation approaches 


Evidence-based trust models often make use of two distinct sources 
of information to evaluate the trustworthiness of a service provider: 
direct trust evidence and indirect trust evidence. The majority of existing 
trust models adopt a weighted average approach when aggregating 
these two sources of trust evidence [3]. Direct trust evidence is often 
assigned a weight of 0 < y < 1, and indirect evidence is assigned a 
corresponding weight of 1 — y. Existing approaches for aggregating di- 
rect and indirect trust evidence can be divided into two broad categories: 
1) static approaches, where the value of y is pre-defined; and 2) dynamic 
approaches, in which the value of y is continually adjusted by the service 
consumer. 

In many papers, static y values for trust evidence aggregation. The 
majority of them tend to take a balanced approach by assigning a 
value of 0.5 to y [6,9,7,11,12]. In some studies, the authors assign the 
value 0 [13,14] or 1 [15] to y to exclusively use only one source of 
trust information. Barber and Kim [16] have empirically shown, without 
considering the presence of biased testimonies, that direct trust evi- 
dence is the most useful to a service consumer over the long term 
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Fig. 1. The general flow of trust-aware interaction decision making for evidence-based trust and reputation management models, and the contributions by the proposed ACT approach. 
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while indirect trust evidence gives an accurate picture more quickly. 
Thus, approaches that discard one source or the other, forfeit some of 
the advantages provided by evidence based trust models. However, 
using a static value for y is also not always a good strategy. 

Some researchers have explored adjusting the value of y dy- 
namically based on different rationales. In [17], the value of y is 
varied according to the number of direct observations on the be- 
havior of a service provider sj available to a service consumer Cj. 
It is assumed that every service consumer starts with no prior in- 
teraction experience with a service provider and gradually accu- 
mulates direct trust evidence over time. Initially, the service 
consumer relies completely on indirect trust evidence (i.e. y = 0) 
to select service providers for interaction. As the number of its interac- 
tions with a service provider s; increases, the value of y also increases 
according to the formula 


i 
eal, if N} < N min (1) 


1 , Otherwise 


where Ni is the total number of direct observations of s;'s behavior by ci, 
and Nmin is the minimum number of direct observations required in 
order to achieve a pre-determined acceptable level of error rate ¢ and 
confidence level ©. Nmin is calculated following the Chernoff Bound 
Theorem [18]: 


1, (1-8 
Nmin = — 5-9! (>). (2) 


This approach is not concerned with filtering potentially biased 
third-party testimonies. Rather, its aim is to accumulate enough direct 
trust evidence so that a service consumer can make a statistically accu- 
rate estimation on the trustworthiness of a service provider without 
relying on indirect trust evidence. Since the value of y increases to 1, 
this approach implicitly assumes that agent behaviors do not change 
with time. This may not always be true and limits the applicability of 
the approach under more dynamic scenarios. On the other hand, the 
ACT approach does not make this assumption and continuously make 
adjustments as the situation changes. 

In [5], an approach based the Q-learning technique [19] to select a y 
value from a pre-specified set of candidate y values has been proposed. 
In order to select appropriate values for this set, expert opinions about 
the underlying system characteristics are assumed to be available. 
Based on the reward accumulated by a service consumer under dif- 
ferent y values, Q-learning selects the y value associated with the 
highest accumulated reward at each time step. This work provided 
the first step towards using interaction outcomes to enable the ser- 
vice consumer to weigh the two sources of trust evidence. However, 
as this method uses a predetermined set of y values, its performance 
is affected by the quality of the expert opinions used to form the set 
of permissible y values. In contrast, the ACT model adjusts both the y 
value as well as the weight values for individual witnesses in finely 
grained steps so that it does not have to rely on the subjective opinions 
of the designer. 


2.2. Testimony filtering approaches 


Anumber of models for filtering potentially biased third-party testi- 
monies have been proposed. However, these models usually assume the 
presence of some infrastructure support or special characteristics in the 
environment. In this section, some representative models in this sub- 
field are discussed. 

The model in [20] makes use of the social relationships among the 
members of a community to determine the credibility of witnesses. 
Pre-determined fuzzy rules are used to estimate the credibility of each 
witness which, in turn, is used as the weight of its testimony for a 


service provider when aggregating all the testimonies. This model relies 
on the availability of social network information among the agents 
which may not be present in many systems. 

In [21], unfair testimonies are assumed to exhibit certain character- 
istics. The proposed approach is closely coupled with the Beta Reputa- 
tion System (BRS) [22] which records testimonies in the form of 
counts of successful and unsuccessful interactions with a service pro- 
vider. The received testimonies are aggregated with equal weights to 
form a majority opinion and then, each testimony is tested to see if it 
is outside the q quartile and (1-q) quartile of the majority opinion. If 
so, the testimony is discarded and the majority opinion updated. This 
model assumes that the majority opinion is always correct. Thus, it is 
not effective in highly hostile environments where the majority of 
witnesses are malicious. 

In [6], it is assumed that the direct experience of the service con- 
sumer is the most reliable source of belief about the trustworthiness 
of a particular service provider, and it is used as the basis for filtering 
testimonies before aggregating them to form a reputation evalua- 
tion. An entropy-based approach is proposed to measure how 
much a testimony deviates from the current belief of the service con- 
sumer before deciding whether to incorporate it into the current be- 
lief. However, by depending on having sufficient direct interaction 
experience with a service provider, this assumption conflicts with 
the purpose for relying on third-party testimonies, which is to help 
service consumers make better interaction decisions when they 
lack direct trust evidence. 

The model in [7] supports interaction outcomes recorded in multi- 
dimensional forms. It applies two rounds of clustering of the received 
testimonies to identify testimonies which are extremely positive or 
extremely negative about a trustee. If neither the extremely positive 
opinion cluster nor the extremely negative opinion cluster forms a 
clear majority, they are both discarded as unfair testimonies and 
the remaining testimonies are used to estimate the reputation of a 
service provider. Otherwise, the majority cluster is considered as 
the reliable testimonies. Due to its iterative nature, the computational 
complexity of this method is high, with a time complexity of O(mn7) 
where m is the number of candidate service providers whose reputa- 
tions need to be evaluated and n is the number of testimonies received 
for each candidate service provider. The method is also not robust 
in hostile environments where the majority of the witnesses are 
malicious. 


3. The ACT approach 
3.1. System model 


Before discussing details of the proposed model, we introduce the 
system model under which the ACT approach is designed to operate. 
At each time step t, a service consumer c; will interact with at most 
one service provider sj in our target system. For each interaction, C; 
chooses a service provider from among several candidates based on 
their estimated trustworthiness values. Whenever c; needs to assess 
the trustworthiness of sj, it draws upon both its own direct trust 
evidence about s; (if there is any) as well as testimonies from a list 
of witnesses W;;(t) which are known by c; at time t to have previously 
interacted with sj. A witness wx may reply to c;'s request at time step t 
with a testimony dk(t). In this study, a malicious wp may distort its 
testimonies before sharing them with others. The service provider 
chosen for interaction by c; at time step t is affected by the selection 
of witnesses as well as the weights given to direct and indirect trust 
evidence. 

Each interaction with s;, c; incurs a utility cost of Cj. If s; successfully 
completes the task assigned to it by c;, c; receives a utility gain of G. We 
assume that the outcome of the interaction O,(t) can be observed by c; 
within the same time step in which the interaction occurs. We further 
assume that the interaction outcome is either successful (O;(t) = 1) 
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or unsuccessful (0;(t) = 0). By comparing the recommendation dk(t) by 
each w,<W;,(t) about s; at time t with O;(t), c; can learn the ranking of 
each w; in Wj(t). New witnesses for discovered by c; over time are 
added into W;. The interaction outcome value, O;j(t), is further com- 
pared with the recommended interaction decision value, D§(t), based 
on direct trust evidence and the value, pir(t), based on indirect trust 
evidence from the testimonies of selected witnesses. Reward and penal- 
ty values are assigned to these two sources of trust evidence by c; in its 
local record to determine how much to rely on either source in the fu- 
ture. In the presence of uncertainty in service providers’ performance 
and the credibility of third-party testimonies, the objective of an indi- 
vidual service consumer is to improve its chance of finding trustworthy 
service providers. 

The general framework of the proposed ACT approach is present- 
ed in Fig. 2. Each service consumer c; keeps two local lists: 1) a list of 
known witnesses, and 2) a list of known service providers. Since a 
witness may only have interacted with a few service providers, the 
list of known witnesses organizes the witnesses into sub-lists 
indexed according to known service providers. The list of known ser- 
vice providers stores the direct trust evidence c; has for each known 
service provider and the weight assigned to the direct trust evidence 
Yyy in the case of that service provider. These two lists grow as c; ac- 
quires more interaction experience with these two types of system 
participants. 

The ACT approach is designed based on a variant of the reinforce- 
ment learning (RL) approach — the actor-critic method [23]. The 
actor-critic method requires minimal computation when selecting 
an action. The actor module represents the policy used to choose 
which witnesses’ testimonies should be selected and how much 
weight each of them should have when aggregating them together 
to form the indirect trust evidence. The policy also determines how 
much weight should be given to the direct and indirect trust evi- 
dence in order to evaluate the service provider's trustworthiness. 
The critic module represents the value function that determines 
whether the service provider c; is better off or worse off after each in- 
teraction with a selected service provider s;. Overtime, the learning 
parameters of the ACT approach are updated in such a way that more 
preference is given to witnesses and the source of trust evidence that 
enhance ¢;'s well-being. 

Although the ACT approach can be used together with many possible 
trust evaluation models, to be specific, we assume that the popular Beta 
Reputation System (BRS) [22] is used as the underlying trust evaluation 


method. The direct trust for s; by c; at t can be calculated using the BRS 
as: 


a+1 


@+B+2 8) 


d 
T(t) ė 


where g is the total number of successful interactions between sj and c;, 
while £ is the total number of unsuccessful interactions between s; and 
c up to t. 


3.2. Learning witness credibility ranking 


According to the principles of reinforcement learning, the ACT 
model needs to define a reward function (rj) for a service consumer 
which, in turn, requires the specification of a source of reinforce- 
ment (u(t). In the critic module, the reward function for c; is defined 
as: 


ry u(t) - (G—G;) = (1-10) - Cj: (4) 


rij is computed at the end of each interaction between c; and sj. The pa- 
rameter u(t) is defined as: 


0, if 0,(t) = O|D,(t) =1 


ni(t) = { 1, if O;(t) = 11Dj(t) =1 = 


Dj,(t) denotes the overall decision by c; to interact with sj at time t 
based on both the direct and indirect trust evidence currently available. 
Here, we only consider the case when the decision is to interact with a 
service provider (i.e. D(t) = 1), because in order for a service consumer 
c; to be able to observe the actual interaction outcome with a service 
provider sj at the end of time t, s; must be selected by c; for interaction 
in that time step. When Dj(t) = 0, it implies that c; deems s; untrustwor- 
thy based on its reputation value. Thus, in these cases, no interaction 
between them will take place at that time and no O;;(t) value can be ob- 
served. In this way, the source of reinforcement is related to the perfor- 
mance of a witness as judged by the actual outcome of an interaction. 
Note that learning only occurs if an interaction takes place. We assume 
that the agents’ direct trust values and indirect trust values are normal- 
ized to a range of [0, 1]. A testimony testk(t) is simply w,’'s direct trust 


i Actor | 

} w1 r(t) Aggregate | 

! testimonies | 

i| w2 72(t, ' 

i (t) from M Indirect Trust , j 

| . F >| Tind(t) | 

H witnesses with Evaluation Ll s1 Reputation1 
i the highest Direct Trust| | E i 

l > i ! : 
wn nw(t) values Evaluation i| s2 | Reputation2 
| Update Update Weights | sN ReputationN 
} Witness , for Trust interaction ' 

} ey eee a Environment 

H Ttw Ttind(t), Ttd| i 

i 7Y ! Select the trustee 
i | with the highest e— 
} ‘ | reputation 

| Update Accumulate Derive Receive | 

i| Learning 4 i¢— Interaction H 

H Reward Reward H 

! | Parameters Outcome ! 

Critic i 


Fig. 2. The general framework of the ACT approach based on reinforcement learning. 
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value for sj based on its own direct trust evidence up to time t. Thus, its 
value is also within the range [0, 1]. 

Once the interaction outcome is known, a reward correction value 
64; is computed for each of the M selected witnesses whose testimonies 
have been used to calculate the reputation of s; namely: 


=F, ldo- (1-0,(6))]. (6) 


Tę; denotes the total number of times that w;'s testimonies about has 
been used by c;, and dk(t) represents the interaction recommendation 
implied by w;'s testimony, test*(t), on sj at time step t and is given by: 


: k 
dt) = 2 9, if testj(t)<Th 7 
i) { Ls otherwise (7) 


where The[0,1] is a predefined threshold value. 6; increases with the 
number of times that wx has given testimonies suggesting a service pro- 
vider is trustworthy but the actual interaction outcome is unsuccessful. 
It is used to penalize the act of unfairly praising a service provider, 
which is the most common form of collusion between service providers 
and witnesses. 

The critic process is carried out by updating the learning parameter 
Pwj for each of the M witnesses whose testimonies resulted in the selec- 
tion of sj by c; at t. This is achieved by jointly considering the latest re- 
ward function value, the accumulated reward, the reward correction 
value, the previous credibility ranking of c;, and the learning rate param- 
eters as follows: 


Pij © Py +P: (ry—fj—ô: by) (1-1). (8) 


The constant (0<p < 1) denotes the learning rate. As p increases, the 
learning parameter pj changes more rapidly as new interaction out- 
comes become available. In this paper, we choose a p value close to 0 
to make pj vary more smoothly. The constant (0 < 6 < 1) represents 
the bias towards penalizing collusion when updating the learning pa- 
rameter; we select its value to be significantly smaller than 1 to avoid 
drastic changes in the value of py. 

The credibility ranking value ™, of each known wx with regard to a 
service provider sj is calculated using the Gibbs softmax method [19] as: 


gP% 
Tki 


ye 


The resulting values of Ty is used to rank the witnesses known to 
ci to facilitate subsequent witness selections. The sum of all m,; values 
always equals to 1. Thus, it can be regarded as the probability of 
soliciting testimonies from each of the witnesses known to c; at 
time t. 

After the credibility ranking values are calculated, the total accumu- 
lated reward rj is updated. It is used as a reference in the process of 
evaluating the well-being of c; resulted from interactions with sj. It is up- 
dated as: 


(9) 


fj Fy + 1—-@) Fy (10) 


where the constant 0 < @ < 1) determines the influence of the latest 

rewards in the smoothed baseline reward rj. When © = 1, only the 

current reward is used to evaluate the credibility of each witness. 
The indirect trust for s; by c; can be computed as the sum of witness 


testimonies weighted by their respective credibility ranking values: 


M k 
ind (p) a ae (ny i testi (t)) 
5 O= M 
die Ty: 


T 


(11) 


3.3. Learning the weights for sources of trust evidence 


With the values of r#(t) and 7j"“(t) calculated using Eqs. (3) and (11) 
respectively, the next step is to aggregate them to compute the reputa- 
tion of s;. In the ACT approach, for each s; known to c;, two critic modules 
are used to learn the weights for the two sources of trust evidence and 
one actor module is used for estimating the trustworthiness of s}. The 
critic module in the proposed method determines the relative merit of 
each source of trust evidence through reward accumulation. The learn- 
ing process is similar to that presented in Section 4.1. Since the two critic 
modules are essentially the same but only use different sources of trust 
evidence as input data, in the following, we only discuss the critic mod- 
ule for the direct trust evidence source. 

For this step, the value function for the direct trust evidence is de- 
signed as: 


ra M(t) -R+ (1—f(t)) -P (12) 
ji(t) = { 1 if O4(t) = DO 
ae) ={ 0, otherwise (13) 


reedi 
pit) — 21, if T(t) >Th 14 
j) { 0, otherwise 4 


ra can be considered as the time averaged per interaction reward 
achieved by c; through relying on its direct trust evidence source 
about s; with the current weight value yj. R and P are predetermined 
constant values for reward and penalty, based on the consequences of 
the interaction decision. The ratio of R to P, rather than their absolute 
values, is important to the learning process. A small R:P ratio means 
that trust is hard for a service provider to gain, but easy to lose. 
The variable ju(t)) determines whether this trust evidence source 
should be rewarded or penalized at time t. Its value toggles between 
0 and 1 according to the relationship between the interaction deci- 
sion DẸ(t), which is related to the direct trustworthiness evaluation 
(0 <7f(t) < 1), and the actual interaction outcome O;;(t). As D§(t) is 
only one component of the overall interaction, it is possible that 
even as D(t) suggests not to interact with sj, the overall decision 
is otherwise. 

Once the latest rg is calculated, it is compared with the baseline re- 
ward f4 accumulated by this trust evidence source to update the learn- 
ing parameter pq according to: 


Pa — Pa +P: (Tata) (1— Ta). (15) 
After pg is updated, f4 is updated to incorporate the latest reward rg: 
Ta = -Ta + (1-@) - Ta. (16) 


Tq can be treated as a basis for comparing whether c; is better off or 
worse off by aggregating the direct trust evidence into the estimation 
for the trustworthiness of sj using the latest y;j value. 

Similarly, the learning parameter pina for the indirect source of trust 
evidence can be obtained. When both pg and Ping are obtained, the 
learning parameters 7g and Tina are updated as: 


A ePa 
T= ePa + e@Pind (17) 
Pina 
A 
Ting = ePa + @Pina ` (18) 


Tq and Tina Can be treated as the probability of selecting each source of 
trust evidence Ta + Tina = 1. In the ACT approach, the value of yj is 
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set to mg. The working process of the ACT approach is shown in 
Algorithm 1. 


Algorithm 1. The ACT testimony aggregation algorithm 


Require: Th (t) for all s; with who c; had prior interactions. 
1: if c; needs to select an SP for interaction then 
2: explorationProbability = random(0, 1) 
3: if explorationProbability < Pr(Exp) then 
4 Randomly select an unknown SP with for interaction 
5: else 
6: Rank known SPs in descending order of their TO values 
7 for each candidate known SP s; do 
8: ci asks the top M ranked witnesses known to c; for testimonies on s; 
9: end for 


10: Evaluate ti"? for each known candidate SP following Eq.(11) 

ii: Evaluate r;(t) for each known SP following Eq.(19) 

12: Delegate the task to the SP with the highest r;(t) value 

13: Observe the interaction outcome O;;(f) with the selected SP 

14: Update T(t) following Eq.(3) 

15: Update the pz; values according to Eq.(8) for all witnesses wą who pro- 
vided testimonies for the selected SP in the last time step 

16: Update their 7%; values following Eq.(9) 

17: Rank known witnesses in descending order of their new 7; values 

18: Update the y;; and (1 — y;;) values according to Eq.(17) and Eq.(18) re- 
spectively 

19: end if 

20: end if 


3.4. Exploration vs. exploitation 


While the strategy for exploiting known witnesses with high credi- 
bility is relatively straightforward (i.e. selecting the top M most credible 
witnesses to request testimonies from), balancing it with exploration 
for addition witnesses requires careful design. In the ACT approach, 
the exploration process is controlled by two parameters: 1) an explora- 
tion probability Pr(Exp), and 2) the magnitude of M. The value of Pr(Exp) 
is initialized to 1 at the start of a service consumer ¢;'s life time to enable 
c; to explore when the list of known witnesses is empty. The value of 
Pr(Exp) is gradually decreased over time until it reaches a pre-defined 
minimum value, Prmin. Testimonies returned by previously unknown 
witnesses are given the benefit of the doubt and included in the calcula- 
tion of the service provider's reputation with weight values equal to the 
lowest 1; among that of the selected known witnesses. This is to ensure 
that c; will always have some opportunity to discover new witnesses. 

A service provider s;'s reputation is calculated as: 


r,(t) 2 yy T(E) + (1—v) TRACE), (19) 


rj(t) represents the overall reputation of sj and is used by c; to estimate 
s;'s trustworthiness. At each time step, c; might have more than one 
candidate service providers to choose from. In this study, we assume 
that c; always selects the service provider with the highest overall repu- 
tation for interaction. For convenience, the symbols used in this paper 
are listed in Table 1. 


4. Experimental evaluations 


In order to comprehensively evaluate the performance of the ACT 
model under different witness behavior conditions, we have designed 
a test-bed which allows the well-being of service consumers adopting 
different approaches to be gauged. Through extensive simulations vary- 
ing the witness population composition, it has been shown that the ACT 
approach significantly outperforms existing approaches in terms of the 
reduction in normalized average utility loss and, in the case of colluding 
witnesses, the reduction in their collusion power. 


Table 1 
Symbols used in this paper. 
Symbol Meaning 
Ci A service consumer. 
QjERY The cost incurred by c; when engaging the service of sj. 
d¥(t)<{0,1} The interaction decision as suggested by testk(t)<[0,1]. 
Dé(t)={0,1} The decision by c; on whether to interact s; with at time t based 
on direct trust evidence only. 
Di™(t)={0,1} The decision by c; on whether to interact s; with at time t based 
on indirect trust evidence only. 
D,(t)={0,1} The overall decision by c; on whether to interact s; with at time 
t based on both direct and indirect trust evidence. 
5€(0,1] The bias towards penalizing collusion. 
GERt The utility derived from a successful interaction. 
ui(t)={0,1} The outcome-based determinant of reward/punishment during 
learning. 
O;(t)={0,1} The outcome of an interaction between c; and sj at time t. 
PERT The penalty assigned to a source of trust evidence. 


PER The learning parameter for updating the credibility ranking of 
the direct trust evidence source. 


PyER The learning parameter for updating the credibility ranking of w, 
with respect to sj. 

Mj > 0 The credibility of wx for s; in c;'s local record. 

Ta = 0 The weight assigned to the direct source of trust evidence by 
a consumer. 

RER The reward assigned to a source of trust evidence. 

r(t)€[0,1] The reputation of sj at time t. 

Ty=R The reward for c; derived from interactions with sj. 

TyER The accumulated reward for c; from past interactions with sj. 

p€(0,1] The learning rate parameter. 

Sj A service provider. 

testi ( t)<[0,1] A testimony from wp with regard to sj at time t. 

TA#(QE[0,1] The direct trust placed on s; by c; at time t. 

zi™(t)<[0,1] The indirect trust placed on s; by c; at time t. 

Wr A witness. 

Oj=[0,1] The reward correction value for wx with regard to sj. 

W;(t) A list of witnesses for s; known to c; at time t. 


4.1. Simulation test-bed 


The test-bed simulates a scenario where a number of service con- 
sumers need the services offered by service providers. A service consum- 
er incurs a cost of C; in order to utilize the service of a service provider. If 
the service provider acts honestly, i.e. satisfies the service consumer's re- 
quest, the service consumer gains an amount of utility of G after the inter- 
action; otherwise, it gains zero utility. Therefore, the maximum average 
utility gain a service consumer can achieve is G — Cj, corresponding to 
all its interactions with service providers being successful; the minimum 
of this value is — Cj, if all its interactions are unsuccessful. 

The main purpose of this test-bed is to investigate the effectiveness 
of the proposed ACT approach in mitigating the adverse effects of unfair 
testimonies relative to existing approaches. Although there are multiple 
ways of modeling the malicious behavior of service providers in a 
system, it is impractical to investigate the proposed model for all possi- 
ble service provider population configurations. In our experiments, we 
adopt one of the common modeling approaches used by previous 
studies such as [9]. The service provider population is hostile to the 
service consumers and consists of 


* 10% honest service providers (which renege randomly with a proba- 
bility of 10%); 

* 10% Type I dishonest service providers (which renege randomly with 
an initial probability of 40%); 

* 40% Type II dishonest service providers (which renege randomly with 
an initial probability of 60%); and 

* 40% Type III dishonest service providers (which renege randomly with 
an initial probability of 80%). 


Except for the honest service provider group, the behavior patterns 
of all other groups changes gradually during the simulation. A service 
provider's behavior can change according to three different profiles: 1) 
increasing reneging probability, 2) decreasing reneging probability, or 


108 H. Yu et al. / Decision Support Systems 66 (2014) 102-113 


3) unchanging reneging probability. The magnitude of each change is 
randomly chosen from the interval [0, 0.01]. Each dishonest service pro- 
vider chooses one of the three profiles in each interaction with equal 
probability (i.e. 1). The test-bed environment consists of 1000 service 
providers with different behavior patterns. During each round of simu- 
lation, each service consumer attempts to solve a total of Nm problems. 
The service consumers select service providers for interaction based on 
their reputation. The outcome of the interaction is assumed to be binary, 
namely successful or unsuccessful, depending on whether the service 
provider provides the requested service. 

There are 100 witnesses who accumulate direct trust evidence about 
the service providers and respond to service consumers requesting tes- 
timonies. When a request for testimony is received by a witness it will 
return a testimony to the requester if it has prior interaction experience 
with the particular service provider in the request; otherwise, it will de- 
cline the request. Two categories of malicious testimony sharing strate- 
gies are studied: 1) random lying, and 2) collusive lying. 

In the case of random lying, a malicious witness does not collude 
with any other service provider. It either positively distorts a testimony 
(ballot-stuffing) or negatively distorts a testimony (badmouthing) 
following a preset lying probability. In the case of collusive lying, a num- 
ber of service providers collude with lying witnesses to inflate their rep- 
utation in the eyes of service consumers (ballot-stuffing). The colluding 
witnesses do not give unfair testimonies about service providers who 
are outside the collusion ring. This is because, relative to a large online 
community, the sizes of collusion rings tend to be small. The costs for 
ballot-stuffing within collusion rings are significantly less than the 
costs for badmouthing a large number of competitors. The actual situa- 
tion observed on e-commerce systems such as eBay.com supports this 
assumption [9]. In both random lying and collusive lying cases, the dis- 
tortions are implemented as offset values added to or subtracted from 
the original testimony. Two types of unfair testimonies are supported 
in the test-bed: 


e Moderately Unfair Testimonies (MUT): the magnitude of the offset is 
randomly chosen in the range [0.1, 0.4]; 

e Highly Unfair Testimonies (HUT): the magnitude of the offset is ran- 
domly chosen in the range [0.8, 1.0]. 


The values of the distorted testimonies are always kept within the 
range [0, 1] by hard-limiting to 1 (or 0) if the distorted testimonies 
after adding (or subtracting) exceeds 1 (or falls below 0). 

In the proposed ACT approach, we use BRS as the trust evaluation 
model in this study. The values selected for the parameters in the ACT 
approach are listed in Table 2. Through these choices of values, we 
give preference for a medium rate of learning and do not allow the latest 
interaction outcome to outweigh past observed behaviors of a service 
provider. They can achieve a good balance between learning speed 
and smooth changes in learning results as suggested by existing litera- 
ture [10,19]. By adopting a well balanced set of values for the learning 
algorithm parameters and varying the behavior patterns of the witness 
agent population, the experiments allow us to draw reasonable conclu- 
sions about the relative performances of the comparative approaches. 


4.2. Evaluation metrics 


Two evaluation metrics from [9] are adopted to facilitate compari- 
sons with state-of-the-art methods: 


1. Normalized Average Utility Leftover (NAUL): the normalized average 
utility (0 < g < 1) measures the average per time step utility gain 


Table 2 

Parameter values used in the simulations. 

Parameter Th Q ô Pp M Nm G G R P Prinin 
Value 05 06 01 04 10 200 5 1 1 —10 01 


as a percentage of the maximum possible utility gain for each service 
consumer over its lifetime. It is calculated as: 


p> 
TN 


3 g(t) — 8min 


§max— Emin 


C= 


(20) 


T is the total number of times a service consumer c; has interacted 
with the service providers, N is the number of service consumers 
adopting the same approach as c; does in the test-bed and Sma, = 
G — Cy, min = — Cy. gi(t) is the actual utility gain of each c; after 
each interaction at time t. If the interaction is successful g;(t) = Zmax; 
otherwise, g;(t) = gmin: NAUL is then (1 — o). With perfect foresight, 
(1 — o) = 0. It measures the percentage difference between the ac- 
tual utility gain and the maximum possible utility gain per service 
consumer per time step (i.e., the leftover utility that the service con- 
sumers following a trust-aware interaction approach are not able to 
gain). The closer (1 — 0) is to 0, the better the performance of a given 
model. 

2. Collusion Power: the Collusion Power, cp, is a measure of the effec- 
tiveness of different models in the face of collusion [9]. It is defined 
as: 


ae #tTy(C;) 


21 
Anl Np (1) 


cp = 


where A, denotes the set of non-colluding service consumers, c; is a 
service consumer in this set, and #try(c;) is the number of times c; 
interacted with any colluding service provider during the simulation. 
In essence, cp represents the percentage of all tasks delegated to any 
of the colluding service providers in the simulated community. 


4.3. Experiment design 


For each experiment, the composition of the common witness pop- 
ulation is altered to simulate different scenarios. In the following sec- 
tions, Hon denotes a population consisting entirely of honest common 
witnesses. BMn denotes a population consisting of n% badmouthing wit- 
nesses and (100 — n)% honest witnesses. BSn denotes a population 
consisting of n% ballot-stuffing witnesses and (100 — n)% honest wit- 
nesses. The malicious witness populations consist of half giving out 
MUTs and half giving out HUTs. 

The experiments include two parts: 


1. Studying the effectiveness of the adaptive trust evidence aggrega- 
tion module of the ACT approach (labeled as ACT”). Under ACT", 
the testimonies from witnesses are treated as of equal importance 
and aggregated through simple averaging. The ACT” module only 
adjusts the weights of the direct trust evidence and the indirect 
trust evidence, y, when calculating the reputation of a candidate 
trustee. 

2. Studying the effectiveness of the ACT approach as a whole (labeled as 
ACT). 


In Part 1 of the experiment, five groups of service consumers are 
simulated for comparison. They are: 


Group y = 0: service consumers who completely rely on indirect trust 
evidence; 

Group y = 0.5: service consumers who rely on a balanced mix of di- 
rect and indirect trust evidence; 

Group y = 1: service consumers who completely rely on direct trust 
evidence; 

Group M2002: service consumers who use the method described in 
[17] to set the y value; 

Group F2007: service consumers who use the method described in [5] 
to set the y value. 
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Methods 


Fig. 3. Ranges of variation of NAUL by service consumer groups under non-collusive 
conditions. 


The group of service consumers equipped with the ACT approach is 
labeled as Group ACT”. Each group consists of 100 agents. All competing 
groups only request for testimonies from the common witness group. 

In Part 2 of this experiment, we compare the performance of the 
complete ACT approach against: 


* Group W2010: service consumers which employ an existing state-of- 
the-art method [9]; 

* Group Y2003: service consumers which employ a classic method [8]; 

* Group B2002: service consumers who only rely on their direct interac- 
tion experience to evaluate a service provider's trustworthiness using 
BRS [22]. 


The group of service consumers equipped with the ACT method is la- 
beled as Group ACT. Each group also consists of 100 agents. All groups 
only request for testimonies from the common witness group same as 
in Part 1 of the experiment. The existing approaches are executed in 
parallel with the proposed model under the same experimental settings 
to obtain fair comparisons of performances. 


4.4. Experiment results — Part 1 


4.4.1. The effect of adaptive y values 

Part 1 of this study is conducted assuming non-collusive common 
witnesses. The common witness population composition is altered 
from BM80 to Hon and then to BS80 to test the performance of service 


NAUL 


—*— F2007 
—e— ACT" 
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Fig. 4. Performance of various service consumer groups under different non-collusive 
common witness populations. 


Table 3 

Improvement of Group ACT” over other groups. 
Group Improvement 
y=0 23.00% 
y=05 16.82% 
y=1 31.99% 
M2002 16.73% 
F2007 15.41% 
Average 20.79% 


consumers employing different testimony aggregation methods. The re- 
sults are summarized in Fig. 3. It can be observed that Group y = 1 
achieves the highest NAUL values as they need more exploration to 
identify trustworthy service providers. Its performance is not affected 
by the changes in the common witness population composition. 
Completely relying on indirect trust evidence is also not a good strategy 
as the performance of Group y = 0 is heavily affected by the presence of 
unreliable witnesses of both BM and BS types. However, the saving in 
exploration from completely relying on third party testimonies allows 
Group y = 0 to achieve lower NAUL values than Group y = 1. Neverthe- 
less, the advantage drops with number of misbehaving witnesses as 
shown in Fig. 4. The performance of the Group y = 0.5 is the best 
among the three groups using static y values. Group F2007's perfor- 
mance is similar to that of Group M2002. As F2007 tries to learn which 
static strategy (y = 0,0.5,or 1) is the best under different conditions, 
its performance more or less tracks that of Group y = 0.5 in our exper- 
iments. Group ACT” outperforms all other methods under all testing 
conditions by an average of 20.79% in terms of the reduction in NAUL. 
A detailed comparison is shown in Table 3. 

The performance achieved by the proposed ACT” service consumers 
can be attributed to their ability to adapt the values of y for each service 
provider as the environment conditions change in a continuous manner. 
Fig. 5 shows a snap-shot of the y value from a service consumer in 
Group ACT” with respect to an honest service provider in its local record. 
As the witness population becomes increasingly hostile, the reliance on 
third-party testimonies is reduced to mitigate their negative influence 
on the service consumer's interaction decisions. 


4.5. Experiment results — part 2 


4.5.1. Performance of ACT under non-collusive lying 

In Part 2 of this study, the performance of the complete ACT ap- 
proach is investigated. The distributions of the NAUL achieved by all 
five models in this study are shown in Fig. 6(a). Group ACT has achieved 


y Value 


0.2; °°: 


o 1 f f 1 i A i ; 
BM80BM60BM40BM20 Hon BS20 BS40 BS60 BS80 
Witness Populations 
Fig. 5. The variation of the y value from the record of a service consumer in Group ACT” 


with respect to an honest service provider under different non-collusive common witness 
populations. 
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(b) Performance of various service con- 
sumer groups under different non-collusive 
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Fig. 6. Results for Experiment Part 2. 


significantly lower level of NAUL than existing models. As shown in 
Fig. 6(b), when the percentage of malicious witnesses increases, the 
performance of Group B2002 is relatively stable as it does not take 
into account testimonies from witnesses when making trustworthiness 


evaluations. However, the NAUL of Group Y2003 deteriorates signifi- 
cantly. The performance of groups W2010 and ACT are relatively consis- 
tent across different witness population configurations. The consistent 
performance achieved by the ACT approach is due to that fact that it 
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Table 4 Table 6 

Improvement of Group ACT over other groups. Improvement of Group ACT over other groups. 
Group Badmouthing Ballot-stuffing Overall Group Collusion power NAUL 
w2010 18.44% 29.91% 25.16% w2010 77.60% 26.37% 
Y2003 64.18% 70.20% 66.98% Y2003 85.94% 55.97% 
B2002 69.07% 74.18% 71.66% 


uses the interaction outcomes with the service providers rather than the 
majority opinion of the witnesses to update the credibility ranking of 
known witnesses, as well as its ability to adjust its preference of the 
two trust evidence sources dynamically. As can be seen from Table 4, 
overall, Group ACT outperforms all other groups in terms of reduction 
in NAUL by significant margins. The advantage is more significant 
under ballot-stuffing conditions due to the addition of the reward cor- 
rection value 0j in Eq. (6) that penalizes positively biased testimonies. 


4.5.2. Performance of ACT under collusive lying 

In our test-bed, the collusive witnesses always form collusion rings 
with Type III malicious service providers to try to promote their reputa- 
tion. The proportion of collusive witnesses in the total common witness 
population is varied from Hon to BS80. From Fig. 6(c), it can be seen 
that the presence of colluding witnesses tricks the Y2003 group into 
interacting more often with collusive service providers than other 
groups. In addition, by comparing Fig. 6(c) with (a), we find that the 
negative impact of collusion is more powerful than that of non- 
collusive random lying. The most adversely affected group is still the 
Y2003 group. The highest NAUL of this group is about 0.3 under BS80 
without collusion. However, under BS80 with collusion, this value 
increases to around 0.6 (as shown in Fig. 6(d)). This is due to the fact 
that colluding witnesses do not give unfair testimonies about non- 
colluding service providers, so that their testimonies are considered 
accurate in these cases. Thus, they are strategically building up their 
credibility with the service consumers in order to mislead them into 
interacting with collusive service providers later. 

The performance of all the models studied in our test-bed deteriorat- 
ed under the influence of collusion as shown in Table 5. Although Group 
ACT and Group W2010 managed to maintain the witness agents' collusion 
power at relatively low levels compared to other groups as illustrated in 
Fig. 6(e), their performances in terms of NAUL still deteriorated under col- 
lusion. It is observed, from Table 6, that the ACT approach significantly 
outperforms all other approaches in terms of mitigating the adverse ef- 
fect of collusion. The over performance in terms of reduction in collusion 
power is the most significant when the majority of the witness popula- 
tion consists of collusive witnesses, as can be seen from Fig. 6(f). 


4.6. Sensitivity analysis 


To study the influence of M on the proposed ACT approach, we alter 
the value of M and re-run the experiments. The value of M is varied to be 
equivalent to between 5% and 20% of the common witness population. 
The experiments are re-run only for the cases where collusion exists 
since collusive testimonies are more powerful in affecting the credibility 
models. From Fig. 7, it can be seen that generally, collusion power in- 
creases with the fraction of colluding witness agents. However, the 
value of collusion power is maintained at a relatively low level by the 
ACT approach. This trend is true for the different values of M. 


Table 5 
Performance deterioration due to collusion (NAUL). 


Group Without collusion With collusion 
ACT 0.0890 0.1825 
w2010 0.1283 0.2479 
Y2003 0.2895 0.4145 


It is expected that the effectiveness of the ACT approach improves 
with M. However, the value of M also determines the storage capacity 
required at each individual service consumer as well as the time taken 
to estimate the reputation of a service provider. Therefore, a service con- 
sumer needs to balance the trade-off between potentially more accurate 
interaction decisions and the extra effort required to gather testimonies 
from more witnesses. 


4.7. Analysis of results 


Several reasons contribute to the superior performance of Group ACT 
model over Groups W2010 and Y2003: 


e Y2003 uses the number of past interactions between a service con- 
sumer c; and the service provider of interest sj to determine whether 
third-party testimonies are required. If the number of past interac- 
tions between c; and s; exceeds a predefined threshold, c; will not 
ask for testimonies when estimating s;'s trustworthiness. However, 
since the behavior of the witnesses are changing in the experiments, 
c;'s direct trust evidence may become outdated. This increases c;'s 
risk exposure in the long run. 

W2010 applies an adaptive strategy in aggregating third-party testi- 
monies. However, it also uses a service consumer c;'s own evaluation 
of a service provider s;'s trustworthiness as a baseline to determine 
which testimonies are potentially unfair. It proposed a measure of 
uncertainty induced by additional testimonies. If a new testimony 
contradicts c;'s current belief about the trustworthiness of s;, it 
would be regarded as increasing c;'s uncertainty and discarded. 
While this approach is more dynamic than Y2003, it still suffers from 
the effect of changing service provider behavior to some degree. 

In contrast, the ACT approach always seeks testimonies from wit- 
nesses when estimating a service provider's reputation. By learning 
the weights assigned to different witnesses’ testimonies based on the 
outcomes after each interaction, the ACT approach dynamically decides 
which witnesses to keep in the top M list for each service provider 
based on their contributions to the well-being of the service consumer. 
Even in the face of highly hostile witness populations, the ACT approach 
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Fig. 7. The influence of the parameter M on the performance of ACT under different 
witness population compositions. 
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still can maintain a relatively good performance by relying more on the 
direct trust evidence source. This mechanism also helps the service con- 
sumers when the behavior of a service provider changes. If this change 
is reflected first in the testimonies, the service consumer can increase 
the weight given to the indirect trust evidence source to reduce the 
need for trial and error; if this change is detected first by the service 
consumer itself, it can increase the weight given to the direct trust ev- 
idence source to reduce its chance of being misled by outdated opinions 
from others. 


5. Implications 


The ACT approach is designed for improving the performance of 
existing reputation models. Such models have been widely applied 
in practical applications. For example, in TripAdvisor,’ individual 
travelers and hotel operators provide reviews on hotels in popular 
destinations to help customers decide which hotels to book. Current- 
ly, customers need to manually assess the content and credibility of 
the reviews and consider their personal experience in the past (if 
there is any) to decide which hotel to book for their next trip. With the 
large volumes of data collected by TripAdvisor (e.g., Hilton Singapore was 
reviewed by more than 1000 people), it is practically impossible for a cus- 
tomer to efficiently consider the available information and make a holistic 
decision. 

The ACT approach can be implemented as a personal trust agent to 
provide decision support for customers in online review systems. The 
agent keeps track of the customer's own reviews on products and 
services using any existing trust model (as long as its trust evalua- 
tions can be normalized to a range of [0,1]). In addition, it filters 
and aggregates reviews for products or services the customer is in- 
terested in and ranks them based on their reputations to advise the 
customer on which one to select. Moreover, the ACT approach elim- 
inates the need for manually tuning the values of parameters impor- 
tant to the performance of underlying trust models. It enables the 
trust models to adapt based on the actual outcomes of past interac- 
tions between a service consumer and other service providers. As a 
result, its performance is less affected by biased third-party testimonies. 
It also does not require additional infrastructure support (e.g., social re- 
lationship information) in order to function. Thus, it forms an efficient 
basis for providing automated decision support to customers of online 
review systems. 


6. Conclusions and future work 


A trust evidence aggregation model, based on the principles of the 
actor-critic learning, was proposed to mitigate the adverse effects of bi- 
ased testimonies. It dynamically adjusts the weights given to selected tes- 
timonies as well as the relative emphasis given to the direct and indirect 
trust evidence sources to reduce a service consumer's risk of being misled 
by biased third-party testimonies or outdated direct past experience. The 
ACT approach can be applied to most existing trust models as long as their 
trust evaluations can be normalized to a range of [0,1] and the interaction 
outcomes can be represented as either successful or unsuccessful. Exper- 
imental results show that ACT outperforms state-of-the-art approaches 
by around 20% in terms of improving the accuracy of finding trustworthy 
service providers in the presence of biased testimonies. 

In the computational trust research literature, the most popular met- 
rics used to determine the relative merits of trust models are individu- 
ally rational in nature. Such measures include various forms of long 
term average monetary gain for service consumers, and the deviation of 
estimated trustworthiness from ground truth. Other means of assessing 
trust decisions on the social welfare of an entire system are rarely 


1 http://www.tripadvisor.com. 


considered. It is our belief that apart from the utility enhancement 
goals, trust models must take into consideration the fair treatment of 
trustworthy service providers during their decision making processes. 
This is a crucial consideration that may hold the key to ensuring the 
long term sustainable operation of a system built on trust. We will inves- 
tigate this topic in subsequent work. 
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